Research Tips: 

Validity and History 


By Dale T. Griffee 


The purpose of this article is to discuss the problem of illusion in 
research, especially in program evaluation. Currently, there are mul- 
tiple research traditions in program evaluation. Nevertheless, all re- 
search orientations are concerned with the role of researchers, for 
example, how they interpret their data and what evidence they offer 
for their interpretation. A more experimental approach tends to be 
used when the researchers are verifying something (they know what 
they are looking for), whereas a more ethnographic approach tends 
to be used when researchers are exploring (they are not clear what 
they are looking for). Ethnographers call the problem of illusion “bias” 
whereas experimental researchers have used the terms “alternative 
hypothesis” or “threats.” This article proceeds from a quasi-experi- 
mental approach and will use the term threat. A threat is any condi- 
tion which blinds or misleads researchers when they interpret their 
results. Although there are many threats in research, a common threat 
for developmental education evaluators is called “history.” 

History can be defined as “events, other than the experimental 
treatment, occurring between pretest and posttest and thus provid- 
ing alternate explanations of effects” (Campbell & Russo, 1999, p. 
80); or to put it another way, things are going on, but the researcher is 
unaware of their influences (Ary, Jacobs, & Razavieh, 1990). 

One example of the threat of history is ESL students making 
friends with native speakers outside of class and improving their En- 
glish simply by conversing with them informally, regardless of the 
teaching they have received in class (Long, 1984). Another example is 
the evaluation of basic writing courses in which posttest scores of an 
experimental class using an innovative curriculum are compared to 
posttest scores of a control class using a traditional curriculum. If the 
mean scores from the experimental classes are higher than the mean 
scores from the control classes, the interpretation of the results is 
that the increase in scores has been caused by the innovation. Is that 
really the case? Could the researcher be unaware of an outside, alter- 
native source of the improvement? For example, the increase in scores 
might be caused by some students going to the college writing center 
or perhaps by taking classes in which the instructor requires writing 
assignments and subsequently helps students with those writing as- 
signments. Either of those situations would constitute a threat to the 
research hypothesis that the innovation was causing the improvement 
in student writing. 

The threat of history is pervasive and dangerous because many 
researchers are often blinded by personal bias, by prejudice, and by 
the condition of frequent ignorance of what is going on in the life of 
students outside of class. The instructors involved in a study may be 
unaware that one of their students has a roommate or a friend who is 
a good writer and tutors the student informally and thereby causes 
the student to receive a high score as a result of this tutoring, which in 
turn raises the average score level. If that student happens to be in the 
experimental class, the higher average scores may be attributed to the 
innovation when, in fact, the innovation is not responsible for the 
score increase. If that student happens to be in the control class, re- 
searchers may think the innovation is not working, when in fact it is. 


This particular prejudice may come into play as a result of an attitude 
that developmental students would never of their own accord seek 
help. “My students, you must be kidding!” Bias may be activated by 
personal infatuation with any current innovation and the strong be- 
lief that this teaching is effective. The threat of history is especially 
powerful because it relies on the implicit and thereby hidden beliefs 
and desires. The way to deal with the threat of history is to make the 
implicit explicit. 

In a strange kind of way, threats can be useful in program evalu- 
ation. Nobody does or can know everything that goes on in the lives 
of these students, but educational investigators can at least think about 
the threat of history before conducting evaluation. Thinking about 
threats is helpful because threats can be anticipated, and thus, at least 
to some extent, can be taken into account before evaluation is initi- 
ated or conducted rather than have threats pointed out by an external 
reviewer after the research is complete and impossible to change or 
even investigate the situation. 

When one first conceives of an evaluation project and thinks 
about a potential research design, the threat of history can be ad- 
dressed by brainstorming possible effects external to the planned in- 
novation. A researcher can ask him or herself as well as others the 
question, “in addition to this innovation and my teaching of it, how 
else might my students be improving their writing outside of class 
unbeknownst to me?” At times it may be important to make a judg- 
ment as to the plausibility of some of the possible actions. Is it plau- 
sible that many students are meeting secretly at night to discuss and 
revise their writing? Perhaps not. Is it plausible that some students 
may be going to the campus writing center? Given that the writing 
center advertises their services, is on campus, accepts manuscripts 
over the internet, and is a free service, perhaps yes. 

In the evaluation on basic writing curriculum mentioned previ- 
ously, two threats were considered: (a) the possibility of students go- 
ing to the writing center and (b) students taking classes that required 
writing for which the instructor was actively helping them. The first 
threat was dealt with by assigning all students, both those in the con- 
trol classes and those in the innovative curriculum classes, to go to 
the writing center. In fact, one meeting for all classes in this evalua- 
tion study was scheduled and held at the writing center. By assigning 
all students to go to the writing center, the threat that some students 
might be going to the center was neutralized. The second threat, that 
some instructors were helping students, was dealt with by interview- 
ing all students to see if that was the case. No such cases were re- 
ported. 

When writing the results of an evaluation, particular threats of 
history may be identified and the actions taken described in detail. If 
space is limited, a brief mention may be made. By thinking about 
history and reporting thoughtful responses to it, research is strength- 
ened and increased in validity. 
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